Continuous deployment pipeline tests

ABSTRACT

In some examples, a system provides a plurality of code changes of a program in a continuous deployment pipeline that performs tests on the plurality of code changes and deploys a code assembly including the plurality of code changes. The system ranks a plurality of tests based on a test attribute and a test context to provide a test rank for each of the plurality of tests, and selects, from among the plurality of tests using the test ranks, a subset of tests to include in a test set for the continuous deployment pipeline. The system executes the test set in the continuous deployment pipeline to test the plurality of code changes.

CROSS REFERENCE TO RELATED APPLICATION

This is a continuation of U.S. application Ser. No. 14/375,005, filedJul. 28, 2014, which is a national stage application under 35 U.S.C.§371 of PCT/US2012/035874, filed Apr. 30, 2012, which are both herebyincorporated by reference in their entirety.

BACKGROUND

Software development life cycles use continuous integration (CI) andcontinuous deployment (CD) to reduce the time code changes spend in aproduction line. Continuous integration automates the process ofreceiving code changes from a specific source configuration management(SCM) tool, constructing deliverable assemblies with the code changes,and testing the assemblies.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting examples of the present disclosure are described in thefollowing description, read with reference to the figures attachedhereto and do not limit the scope of the claims. In the figures,identical and similar structures, elements or parts thereof that appearin more than one figure are generally labeled with the same or similarreferences in the figures in which they appear. Dimensions of componentsand features illustrated in the figures are chosen primarily forconvenience and clarity of presentation and are not necessarily toscale. Referring to the attached figures:

FIG. 1 illustrates a network environment to prioritize a plurality oftests in a continuous deployment pipeline according to an example;

FIG. 2 illustrates a block diagram of an apparatus to prioritize aplurality of tests in a continuous deployment pipeline according to anexample;

FIG. 3 illustrates block diagram of the system of FIG. 1 according to anexample;

FIG. 4 illustrates a block diagram of a computer readable medium useablewith a system, according to an example;

FIG. 5 illustrates a schematic diagram of a process that prioritizes aplurality of tests in a continuous deployment pipeline according to anexample; and

FIGS. 6-7 illustrate flow charts of methods to prioritize a plurality oftests in a continuous deployment pipeline according to examples.

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and in which isillustrated by way of specific examples in which the present disclosuremay be practiced. It is to be understood that other examples may beutilized and structural or logical changes may be made without departingfrom the scope of the present disclosure.

Continuous integration (CI) and continuous deployment (CD) automate theconstruction, testing, and deployment of code assemblies with a codechange.

Continuous integration automates the process of retrieving code changesfrom the SCM tool, constructing deliverable assemblies, such asexecuting a build and unit testing the assemblies. The automation beginsafter a code change is committed to a source configuration management(SCM) tool. When the code change is committed to the SCM tool, the codechange is assigned to a particular continuous deployment pipeline (CDpipeline or deployment pipeline). The code change moves through thecontinuous deployment pipeline as the code change is tested as part of acode assembly.

Continuous deployment extends continuous integration by automaticallydeploying the assemblies into a test environment and executing testingon the assemblies. The amount of testing is determined by the continuousdeployment pipeline. The test sets used for the continuous deploymentpipelines are typically static test sets assigned based on theclassification of the continuous deployment pipeline, such as lowpriority or high priority. Typical test sets include unit tests andapplication programming interface tests. User interface automation testsare rarely included in the test sets due to the execution time beinglonger than unit tests and application programming interface tests. As aresult, the test sets lack a quality guarantee since the test sets donot include user interface tests. Moreover, the test set may be manuallyset-up to include the user interface tests; however, manual changes totest sets does not guarantee consistency and requires manual operations.

In examples, a method to prioritize a plurality of tests in a continuousdeployment pipeline is provided. The method ranks the plurality of testsbased on a test attribute and a test context to provide a test rank foreach of the plurality of tests. The method sets a test set for thecontinuous deployment pipeline using the test ranks. The method executesthe test set in the continuous deployment pipeline.

The phrase “code change” refers to a change in the source code for anysoftware application. The phrase code change may also refer to a codechange that is part of a code assembly constructed as part of acontinuous integration process.

The phrase “continuous deployment pipeline” or “deployment pipeline”refers to a set of actions executed serially and/or in parallel on aqueue of code changes. For example, the continuous deployment pipelinemay include building the code, executing unit tests, deploying the code,running automated tests, staging the code, running end-to-end tests, anddeploying the code to production. Each continuous deployment pipelinemay be classified to receive code changes that match a defined set ofcriteria, for example, a specific continuous deployment pipeline may beused for low risk and high priority code changes. The test set used totest the pipeline may similarly be aligned with the defined set ofcriteria for the code changes.

The phrase “test set” refers to the tests run on a continuous deploymentpipeline in a simulated environment. The test set tests functionalityand/or identifies deficiencies of the application under test (AUT). Thetest set may include unit tests to test integration of the code changesand/or functional tests with the code change, such as applicationprogramming interface (API) or user interface (UI) tests.

The phrase “test attribute” refers to data associated with tests andprevious executions of tests.

The phrase “test context” comprises at least one of a code change, abusiness priority, and a test value. The test value may include theability of the test to detect defects, the importance of defects foundin a test, and the dependency of the test on other tests.

The phrase “test rank” or “test ranking” refers to a ranking or valueassigned to a test based on one or more criteria. The criteria may bedefined based on codes changes, test characteristics, businesspriorities, and/or continuous deployment pipelines associated with thetest from prior executions of the test set.

FIG. 1 illustrates a network environment 100 according to an example.The network environment 100 includes a link 10 that connects a testdevice 12, a deployment device 14, a client device 16, and a data store18. The test device 12 represents generally any computing device orcombination of computing devices that test a plurality of code changesfrom a deployment device 14. The deployment device 14 represents acomputing device that receives the code changes and deploys code changesin the continuous deployment pipeline.

The client device 16 represents a computing device and/or a combinationof computing devices configured to interact with the test device 12 andthe deployment device 14 via the link 10. The interaction may includesending and/or transmitting data on behalf of a user, such as the codechange. The interaction may also include receiving data, such as asoftware application with the code changes. The client device 16 may be,for example, a personal computing device which includes software thatenables the user to create and/or edit code for a software application.

The test device 12 represents a computing device and/or a combination ofcomputing devices configured to prioritize a plurality of tests in acontinuous deployment pipeline. The test device 12 is also configured toexecute a set of tests or test sets on the continuous deploymentpipeline in an application under test environment to integrate theplurality of code changes for use in a software application.

The test set and/or the code changes may be stored in the data store 18.The data store 18 represents generally any memory configured to storedata that can be accessed by the test device 12 and the deploymentdevice 14 in the performance of its function. The test device 12functionalities may be accomplished via the link 10 that connects thetest device 12 to the deployment device 14, the client device 16, andthe data store 18.

The link 10 represents generally one or more of a cable, wireless, fiberoptic, or remote connections via a telecommunication link, an infraredlink, a radio frequency link, or any other connectors or systems thatprovide electronic communication. The link 10 may include, at least inpart, an intranet, the Internet, or a combination of both. The link 10may also include intermediate proxies, routers, switches, loadbalancers, and the like.

FIG. 1, further illustrates a block diagram of the test device 12 toprioritize a plurality of tests in a continuous deployment pipeline. Thetest device 12 includes a test engine 22 and a decision engine 24. Thetest engine 22 represents generally a combination of hardware and/orprogramming that execute the test set in the continuous deploymentpipeline. The decision engine 24 represents generally a combination ofhardware and/or programming that rank the plurality of tests based on atest attribute and a test context to provide a test rank for each of theplurality of tests. The decision engine 24 also sets a test set for thecontinuous deployment pipeline using the test ranks.

FIG. 2 illustrates a block diagram of an apparatus 200 to prioritize aplurality of tests in a continuous deployment pipeline according toexamples. The apparatus 200 is useable with a continuous deploymentpipeline and includes the test device 12, a memory 26, and a processor28. The test device 12 prioritizes a plurality of tests in thecontinuous deployment pipeline. The memory 26 stores a set ofinstructions. The processor 28 is coupled to the memory 26 to executethe set of instructions. The set of instructions set a test set for thecontinuous deployment pipeline based on a ranking of each of theplurality of tests. The ranking is determined using a test attribute anda test context. The set of instructions also execute the test set in thecontinuous deployment pipeline and update the test attribute and thetest context based on a test result from the execution of the test. Thetest result including data corresponding to at least one of the testattribute and the test context.

FIG. 3 illustrates a block diagram of the system 100 of FIG. 1 accordingto a further example. The system 100 includes the test device 12, thedeployment device 14 and the data store 18. As illustrated in FIG. 1,the test device 12, the deployment device 14, and the data store 18 areconnected via the link 10.

The data store 18 may store data accessible by the test device 12 and/orthe deployment device 14. The data store 18 is, for example, a databasethat stores, a code change 31, a test 32 (or plurality of tests), a testranking 33 (or test rank), a test set 35, a test attribute 36, a testcontext 37, a weighted value 38, and a test result 39.

The test device 12 is illustrated as including a test engine 22 and adecision engine 24. The test device 12 is connected to the deploymentdevice 14, which receives the code change 31 from the client device 16.The code change 31 is tested in the test device 12 using the tests 32that have test rankings 33. The test device 12 determines a set of testsor test sets 35 to use in the continuous deployment pipeline during thetesting.

The test engine 22 represents generally a combination of hardware and/orprogramming to execute the test set 35 in a continuous deploymentpipeline. The test engine 22 receives an instruction and/or the testsets 35 from the decision engine 24 to execute the test set 35.

The decision engine 24 represents generally a combination of hardwareand/or programming that ranks the plurality of tests 32 and sets a testset 35 for the continuous deployment pipeline. The decision engine 24ranks the plurality of tests 32 based on a test attribute 36 and a testcontext 37 to provide a test ranking 33 for each of the plurality oftests 32. The decision engine 24 also assigns the test attribute 36 andthe test context 37 a weighted value 38. The weighted value 38 isdetermined based on the continuous deployment pipeline associatedtherewith. The test attribute 36 and test context 37 are collected asdata inputs from multiple sources and are compounded with a weightedvalue 38 for each continuous deployment pipeline as performance of thetests 32 are requested.

The test attribute 36 comprises data associated with the plurality oftests 32 and previous executions of the plurality of tests 32. Examplesof test attributes 36 include the following a test type, such as asystem, function, or performance test. The impact that the test typewill have on the rank varies depend on the continuous deploymentpipeline. For example a comprehensive continuous deployment pipeline mayselect a single test for each test type, while a quick continuousdeployment pipeline may exclude one or more types of tests 32, such as aperformance test. Another type of test attribute 36 is the coverage ofthe application under test (AUT) areas. The coverage of the AUT areasthat have a positive impact on test ranking 33 or results in a highertest ranking 33 when the test yields more coverage. Test fixingfrequency is another test attribute 36. Test fixing frequency can have anegative impact on the test ranking 33, as the more frequent a test 32needs to be fixed the less stable and less valuable the test 32. Anothertest attribute 36 is test cost, such as costs in resources, for exampleCPU, memory, network, and storage. The more the test costs in resourcesand/or actual price, such as for cloud-based deployments, the lower thetest ranking 33. The last execution time of the test is another type oftest attribute 36. For example, the older the last execution time is thehigher the test ranking 33. A final example of the test attribute 36 isaverage execution time, which may result in a test 32 having a lowertest ranking 33 because it takes more time to execute and uses more timein the pipeline, which may reduce the total number of tests 32 executed.

The test context 37 comprises at least one of a code change 31, abusiness priority, and a test value. The test value may include, forexample, a value corresponding to the ability of test to detect defects,the importance of defects the test finds, and the dependency of tests 32on other tests 32. Examples of application lifecycle management testcontexts 37 are provided as follows. A test context 37 includes a recentcode change 31 linked to the test via requirements and defects. The testcontext 37 receives a higher test ranking 33 when the test verifies alarge number of code changes 31 recently committed. Another test context37 is the average or sum of the business priorities linked to the testvia the requirements. The higher the business priority the higher testranking 33. Moreover, the test context 37 in this case may be used tofilter out no-critical business requires in specific business-criticalcontinuous deployment pipelines. Another test context 37 is anassessment of the ability of the test to find a defect or the averagenumber of defects found by the test per test execution. The more defectsthe test 32 finds the higher the test ranking 33.

Similarly, the average severity and/or importance of the defect found bythe test also results in a higher test ranking 33. A final example ofthe test context 37 is a test's 32 dependency on other tests 32. Forexample, a comprehensive continuous deployment pipeline may selectinter-dependent tests 32 as test groups, while quick continuousdeployment pipelines may avoid inter-dependent test to minimizeexecutions and application under test deployments.

The decision engine 24 also sets a test set 35 for the continuousdeployment pipeline using the test ranks 33. The decision engine 24 setsthe test set 35 using at least one of the following criteria associatedwith the plurality of tests 32: an execution time, a business value, anda comprehensiveness value. Additional criteria that may be used todetermine the test ranking 33 and/or the test set 35 as defined by thedeployment device 14, such as costs and resources. For each continuousdeployment pipeline, the test rank 33 is computed for all the tests 32and weights may be associated with the test ranks 33 based on thecharacteristics of the continuous deployment pipeline. The tests 32 maythen be classified and stored with test rankings 33 and weighted values38 that are easily accessible by the decision engine 24 when thecontinuous deployment pipeline is executing and the test set 35 isrequested by the test device 12 and/or the deployment device 14.

After the continuous deployment pipeline is established, the decisionengine 24 sets the test set 35 to meet the criteria of the specificcontinuous deployment pipeline being executed and provides the test set35 to the test engine 22. The test set 35 may meet the criteria, such astime, cost, and/or application coverage level criteria, of the specificcontinuous deployment pipeline, which includes obtaining an optimized ormaximal aggregate rank of the tests 32. In meeting the criteria, thetest set 35 may not include a high-ranking test (test ranking 33) inorder to include a lower-ranking test (test ranking 33) that provides anaggregate rank desired.

Furthermore, the decision engine 24 collects a test result 39 from theexecuted test set 35. The test result 39 including data corresponding toat least one of the test attribute 36 and the test context 37. Forexample, the test result 39 may include new found defects and executiontimes for a test 32. The decision engine 24 updates the test attribute36 and the test context 37 based on a test result 39 from the executionof the test set 35. Moreover, the decision dynamically collects andautomatically updates the test attribute 36 and the test context 37based on execution of the test set 35. The test engine 22 maysimultaneously collect the test result 39 while executing the test set35. Simultaneous executing the test set 35 and collection of the testresults 39 including the test attributes 36 and test context 37 data todetermine the test rankings 33 and weighted values 38 may be performedbased on the capabilities of the processor and/or computing resources.Moreover, the test rankings 33 and weighted values 38 may also beupdated simultaneously with the execution of the test set 35.

The deployment device 14 includes a deployment engine 34. The deploymentengine 34 represents generally a combination of hardware and/orprogramming that deploys the code change 31 after testing in anapplication under test environment. The deployment device 14 deploys thetested code change 31 via a deployment engine 34. The deployment engine34 may work together with the test engine 22 and the decision engine 24to execute the test sets 35. Moreover, the deployment engine 34 controlsa continuous deployment pipeline after the code assembly with the codechanges 31 pass the test sets 35.

FIG. 4 illustrates a block diagram of a computer readable medium useablewith the system 100 of FIG. 1 according to an example. In FIG. 4, thetest device 12 is illustrated to include a memory 26, a processor 28,and an interface 43. The processor 28 represents generally any processorconfigured to execute program instructions stored in memory 26 toperform various specified functions. The interface 43 representsgenerally any interface enabling the test device 12 to communicate withthe deployment device 14 via the link 10, as illustrated in FIGS. 1 and3.

The memory 26 is illustrated to include an operating system 44 andapplications 45. The operating system 44 represents a collection ofprograms that when executed by the processor 28 serve as a platform onwhich applications 45 may run. Examples of operating systems 43 includevarious versions of Microsoft's Windows® and Linux®. Applications 45represent program instructions that when executed by the processor 28function as an application that prioritizes a plurality of tests 32 in acontinuous deployment pipeline. For example, FIG. 4 illustrates a testmodule 46 and a decision module 47 as executable program instructionsstored in memory 26 of the test device 12.

Referring back to FIGS. 1-3, the test engine 22 and the decision engine24 are described as combinations of hardware and/or programming. Asillustrated in FIG. 4, the hardware portions may include the processor28. The programming portions may include the operating system 44,applications 45, and/or combinations thereof. For example, the testmodule 46 represents program instructions that when executed by aprocessor 28 cause the implementation of the of the test engine 22 ofFIGS. 1-3. The decision module 47 represents program instructions thatwhen executed by a processor 28 cause the implementation of the of thedecision engine 24 of FIGS. 1-3.

The programming of the test module 46 and decision module 47 may beprocessor executable instructions stored on a memory 26 that includes atangible memory media and the hardware may include a processor 28 toexecute the instructions. The memory 26 may store program instructionsthat when executed by the processor 28 cause the processor 28 to performthe program instructions. The memory 26 may be integrated in the samedevice as the processor 28 or it may be separate but accessible to thatdevice and processor 28.

In some examples, the program instructions may be part of aninstallation package that can be executed by the processor 28 to performa method using the system 100. The memory 26 may be a portable mediumsuch as a CD, DVD, or flash drive or a memory maintained by a serverfrom which the installation package can be downloaded and installed. Insome examples, the program instructions may be part of an application orapplications already installed on the server. In further examples, thememory 26 may include integrated memory, such as a hard drive.

FIG. 5 illustrates a schematic diagram 500 of the process thatprioritizes a plurality of tests 32 in a continuous deployment pipelineaccording to an example. FIG. 5 illustrates the functions of the testdevice 12 and the deployment device 14.

The deployment device 14 receives a code change 31 from a sourceconfiguration management (SCM) tool 52 and business criteria 53 from anapplication lifecycle management (ALM) tool 54. The business criteria 53refers to business factors that are used to assign code changes to acontinuous deployment pipeline using, for example, filtering rules. Thebusiness criteria 53 may correspond to data associated with theapplication being modified and/or the code changes, such as author of acode change, number of lines of code in the code change, and/or numberof files changed The deployment device 14 assigns the code change 31 toa continuous deployment pipeline 50, such as CD pipeline A 50A or CDpipeline B 50B.For example, CD pipeline 50A may be a high prioritypipeline 50A for code changes 31 that are determined by the deploymentdevice 14 to be tested and deployed quickly. Similarly, CD pipeline 50Bmay be a normal or low priority pipeline. The deployment device 14provides a deploy to test 55 instruction to the test device 12 toinitiate performance of the tests 56.

The test device 12 uses the decision engine 24 to ranks the plurality oftests 32 and sets a test set 35 for the continuous deployment pipeline50. The decision engine 24 uses a test attribute 36 and a test context37 to rank the plurality of tests 32. The decision engine 24 may use avariety of test attribute 36 and test context 37 data to rank the tests32 and determine the test set 35. Additional data that may be consideredto determine the test rankings 33 include, for example, the code change31, the plurality of tests 32, the weighted value 38, and the testresults 39. The ranking of the tests 32 for each pipeline and even eachcode assembly with codes changes 31 are automated and may vary, as therank specific to the pipeline, the application, and the types of codechanges 31. Moreover, the test set 35 may be determined by the decisionengine 24 to optimize the tests 32 performed based on the test rankings33.

After the test set 35 is determined, the test set 35 is provided to thetest engine 22, which executes the test set 35 in the continuousdeployment pipeline 50 in a simulated or AUT environment. For example,CD pipeline A 50A uses test set A 57, and CD pipeline B 50B uses testset B 57. Depending on the continuous deployment pipeline 50 the testsets may be the same or distinct. The test set 35 execution isillustrated in the perform tests 56 portion of the continuous deploymentpipeline 50. The code changes 31 remain in the respective continuousdeployment pipeline 50 until the assembly with the code change 31 passesthe test set 35 determined by the test device 12. The test results 59are sent to the decision engine 24 either when all the testing iscomplete and/or simultaneously during the testing. The test results 59may include test attribute 36 and test context 37 data that isdynamically and automatically collected and complied by the decisionengine 24. The test results 59 are then automatically used to determinefuture test rankings 33 and test sets 35.

After the code changes 31 pass the performance tests 56, including theexecution of the test set 35, the deployment device 14 may deploy theassembly with the code changes 31 to production 58 and/or releases asoftware application with the code change 31.

FIG. 6 illustrates a flow diagram 600 of a method, such as a processorimplemented method to prioritize a plurality of tests in a continuousdeployment pipeline. In block 62, the plurality of tests are rankedbased on a test attribute and a test context 37 to provide a test rankfor each of the plurality of tests. The test attribute includes dataassociated with the plurality of tests 32 and previous executions of theplurality of tests. Examples of the test attribute include such as testtype, coverage of AUT areas, last execution time, test fixing frequency,test cost, and/or average execution time. The test context 37 includesat least one of a code change, a business priority, and a test value.The test value may include, for example, a value corresponding to theability of test to detect defects, the importance of defects the testfinds, and the dependency of tests on other tests. Other examples ofapplication lifecycle management test contexts include the following: arecent code change linked to the test via requirements and defects,average or sum of business priorities linked to the test, defect finingassessment, an average severity or importance of defects, and/ordependency of test on other tests 32.

The test set for the continuous deployment pipeline is set in block 64using the test ranks. The test attribute and the test context may alsobe assigned a weighted value. The weighted value is determined based onthe continuous deployment pipeline associated therewith. Moreover, thetest attribute and the test context are dynamically collected andautomatically updated based on execution of the test set.

In block 66, the test set is executed in the continuous deploymentpipeline. Based on the execution of the test set, a test result iscollected. The test result includes data corresponding to at least oneof the test attribute and the test context. The test result may besimultaneously collected during the execution of the test set. The testresult is then used to update the test attribute and the test contextbased on the test result from the execution of the test set.

FIG. 7 illustrates a flow diagram 700 of a method, such as a processorimplemented method, to prioritize a plurality of tests in a continuousdeployment pipeline. In block 72, a test set for the continuousdeployment pipeline is set based on a ranking of each of the pluralityof tests. The ranking is determined using a test attribute and a testcontext. The test set is executed in the continuous deployment pipelinein block 74. The test attribute and the test context are updated inblock 76. The update is based on a test result from the execution of thetest. The test result includes data corresponding to at least one of thetest attribute and the test context.

FIGS. 1-7 aid in illustrating the architecture, functionality, andoperation according to examples. The examples illustrate variousphysical and logical components. The various components illustrated aredefined at least in part as programs, programming, or programinstructions. Each such component, portion thereof, or variouscombinations thereof may represent in whole or in part a module,segment, or portion of code that comprises one or more executableinstructions to implement any specified logical function(s). Eachcomponent or various combinations thereof may represent a circuit or anumber of interconnected circuits to implement the specified logicalfunction(s).

Examples can be realized in any computer-readable media for use by or inconnection with an instruction execution system such as acomputer/processor based system or an ASIC (Application SpecificIntegrated Circuit) or other system that can fetch or obtain the logicfrom computer-readable media and execute the instructions containedtherein. “Computer-readable media” can be any media that can contain,store, or maintain programs and data for use by or in connection withthe instruction execution system. Computer readable media can compriseany one of many physical media such as, for example, electronic,magnetic, optical, electromagnetic, or semiconductor media. Morespecific examples of suitable computer-readable media include, but arenot limited to, a portable magnetic computer diskette such as floppydiskettes or hard drives, a random access memory (RAM), a read-onlymemory (ROM), an erasable programmable read-only memory, or a portablecompact disc.

Although the flow diagrams of FIGS. 6-7 illustrate specific orders ofexecution, the order of execution may differ from that which isillustrated. For example, the order of execution of the blocks may bescrambled relative to the order shown. Also, the blocks shown insuccession may be executed concurrently or with partial concurrence. Allsuch variations are within the scope of the present invention.

The present disclosure has been described using non-limiting detaileddescriptions of examples thereof and is not intended to limit the scopeof the present disclosure. It should be understood that features and/oroperations described with respect to one example may be used with otherexamples and that not all examples of the present disclosure have all ofthe features and/or operations illustrated in a particular figure ordescribed with respect to one of the examples. Variations of examplesdescribed will occur to persons of the art. Furthermore, the terms“comprise,” “include,” “have” and their conjugates, shall mean, whenused in the present disclosure and/or claims, “including but notnecessarily limited to.”

It is noted that some of the above described examples may includestructure, acts or details of structures and acts that may not beessential to the present disclosure and are intended to be exemplary.Structure and acts described herein are replaceable by equivalents,which perform the same function, even if the structure or acts aredifferent, as known in the art. Therefore, the scope of the presentdisclosure is limited only by the elements and limitations as used inthe claims.

What is claimed is:
 1. A method comprising: providing, by a systemcomprising a processor, a plurality of code changes of a program in acontinuous deployment pipeline that performs tests on the plurality ofcode changes and deploys a code assembly including the plurality of codechanges; ranking, by the system, a plurality of tests based on a testattribute and a test context to provide a test rank for each of theplurality of tests; selecting, by the system from among the plurality oftests using the test ranks, a subset of tests to include in a test setfor the continuous deployment pipeline; and executing, by the system,the test set in the continuous deployment pipeline to test the pluralityof code changes.
 2. The method of claim 1, wherein the selecting isaccording to a criterion relating to execution time of performing testsin the continuous deployment pipeline.
 3. The method of claim 1, whereinthe selecting is according to a criterion relating to a cost ofperforming tests in the continuous deployment pipeline.
 4. The method ofclaim 1, further comprising keeping the plurality of code changes in thecontinuous deployment pipeline until the code assembly with theplurality of code changes passes the test set.
 5. The method of claim 1,further comprising receiving, in the continuous deployment pipeline, theplurality of code changes that match a specified criterion of thecontinuous deployment pipeline.
 6. The method of claim 1, wherein thecontinuous deployment pipeline is a first continuous deploymentpipeline, the method further comprising: receiving, in a secondcontinuous deployment pipeline, a further plurality of code changes thatmatch a specified criterion of the second continuous deploymentpipeline; ranking a further plurality of tests based on a test attributeand a test context to provide a test rank for each of the furtherplurality of tests; selecting, from among the further plurality of testsusing the test ranks for the further plurality of tests, a subset oftests to include in a second test set for the second continuousdeployment pipeline; and executing the second test set in the secondcontinuous deployment pipeline to test the further plurality of codechanges.
 7. The method of claim 1, further comprising: collecting a testresult from the executed test set, the test result including datacorresponding to at least one of the test attribute and the testcontext.
 8. The method of claim 1, wherein the test attribute comprisesdata associated with the plurality of tests and previous executions ofthe plurality of tests.
 9. The method of claim 1, wherein the testcontext comprises the plurality of code changes.
 10. The method of claim1, wherein the test attribute and the test context are each assigned aweighted value is determined based on the continuous deploymentpipeline.
 11. A system to prioritize a plurality of tests in acontinuous deployment pipeline, the system comprising: a processor; anda non-transitory storage medium storing instructions executable on theprocessor to: receive a plurality of code changes of a program in acontinuous deployment pipeline that performs tests on the plurality ofcode changes and deploys a code assembly including the plurality of codechanges; rank a plurality of tests based on a test attribute and a testcontext to provide a test rank for each of the plurality of tests;select, from among the plurality of tests using the test ranks, a subsetof tests to include in a test set for the continuous deploymentpipeline; and execute the test set in the continuous deployment pipelineto test the plurality of code changes.
 12. The system of claim 11,wherein the test attribute comprises data associated with the pluralityof tests and previous executions of the plurality of tests.
 13. Thesystem of claim 11, wherein the test context comprises the plurality ofcode changes.
 14. The system of claim 11, wherein the selecting isaccording to a criterion relating to execution time of performing testsin the continuous deployment pipeline.
 15. The system of claim 11,wherein the selecting is according to a criterion relating to a cost ofperforming tests in the continuous deployment pipeline.
 16. The systemof claim 11, wherein the instructions are executable on the processor tokeep the plurality of code changes in the continuous deployment pipelineuntil the code assembly with the plurality of code changes passes thetest set.
 17. The system of claim 11, wherein the instructions areexecutable on the processor to receive, in the continuous deploymentpipeline, the plurality of code changes that match a specified criterionof the continuous deployment pipeline.
 18. A non-transitorymachine-readable storage medium storing instructions that upon executioncause a system to: receive a plurality of code changes of a program in acontinuous deployment pipeline that performs tests on the plurality ofcode changes and deploys a code assembly including the plurality of codechanges; rank a plurality of tests based on a test attribute and a testcontext to provide a test rank for each of the plurality of tests;select, from among the plurality of tests using the test ranks, a subsetof tests to include in a test set for the continuous deploymentpipeline; and execute the test set in the continuous deployment pipelineto test the plurality of code changes.
 19. non-transitorymachine-readable storage medium system of claim 15, wherein theinstructions are executable on the processor to keep the plurality ofcode changes in the continuous deployment pipeline until the codeassembly with the plurality of code changes passes the test set.
 20. Thenon-transitory machine-readable storage medium of claim 15, wherein theinstructions are executable on the processor to receive, in thecontinuous deployment pipeline, the plurality of code changes that matcha specified criterion of the continuous deployment pipeline.